Sample Average Approximation with Sparsity- Inducing Penalty for High-Dimensional Stochastic Programming

The theory on the traditional sample average approximation (SAA) scheme for stochastic programming dictates that the number of samples should be polynomial in the number of problem dimensions in order to ensure proper optimization accuracy. In this paper, we study a modification to SAA in the scenario where the global minimizer is either sparse or can be approximated by a sparse solution. By making use of a regularization penalty referred to as the folded concave penalty (FCP), we show that the required number of samples can be significantly reduced: the sample size is only required to be poly-logarithmic in the number of dimensions. As an immediate implication of our result, a flexible class of folded concave penalized sparse M-estimators in high-dimensional statistical learning may yield a sound performance even H. Liu Department of Radiation Oncology, Stanford University, Stanford, CA 94305, USA E-mail: [email protected] X. Wang Harold and Inge Marcus Department of Industrial and Manufacturing Engineering, The Pennsylvania State University, University Park PA 16802, USA E-mail: [email protected] T. Yao Harold and Inge Marcus Department of Industrial and Manufacturing Engineering, The Pennsylvania State University, University Park PA 16802, USA E-mail: [email protected] R. Li Department of Statistics, The Pennsylvania State University, University Park PA 16802, USA E-mail: [email protected] Y. Ye Department of Management Science and Engineering, Stanford University, Stanford, CA 94305, USA E-mail: [email protected] 2 Hongcheng Liu et al. when the problem dimension cannot be upper-bounded by any polynomial function of the sample size.